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(54) Method and system for Improving color images 



(57) A method and system for Improving a color im- 
age separable into three color components. A noise-fil- 
tering unit is used to filter the color components for pro- 
viding noise-filtered color components, The edge Infor- 
mation extracted from one color component Is used to 



adjust the noise-filtered color component of a different 
color. The algorithm for noise filtering of color compo- 
nents can be based on linear spatial filters, non-linear 
filters or the combinations thereof. The algorithm for 
edge detection can be based on directional or non-dl- 
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Description 

Field of the Invention 

* [Q001] The present Invention relates generally to Image processing and, more particularly, to noise filtering a color 
image. 

Background of the Invention 

10 [0002] Noise reduction in image processing Is widely known, and methods for noise reduction are abundant. In the 
past, different noise reduction methods were often adapted to filter out a specific type of noise. For example, non-linear 
filters, such as median filters, are well suited for filtering out "shot noise', which affects the pixel value of individual 
pixels in a random fashion, whereas llnoarf liters, such as mean filters, are better suited lor filtering out Gaussian noise. 
These filters are, in effect, low-pass filters to be used to blur the image. Thus, the sbe of the pixel array, or stencil, of 

is the filters is generally kept small so that the blurring does not significantly degrade the image. 

[0003] With the advent of digital cameras, a new type of pixel sampling scheme (Bayer matrix) has become popular, 
whare only a subset of the pixels is sampled for each color component. Before the Images takBn through a Bayer 
matrix can be used, the color components- have to be up-sampled by means of Inteipolation. The foimat of images 
taken through a Bayer matrix is shown in Figure 1 . When color components are separated from the Bayer matrixed 

so image as shown, each color component has many missing pixels and the pixel locations of one color component are 
different from the pixel locations of another color component. As shown in Figure 1 , one half of the pixel lines in a Bayer 
matrixed image contain blue color pixels interlaced with green color pixels, and the other half of the pixel lines contain 
red color pixels Interlaced with green color pixels. Thus, the missing pixels In the color components are 50% In the 
green component, and 75% In either the red or blue component. The interpolation of each color component, known 

» as color filter array (CFA) interpolation or de-mosaic interpolation, smears the noise from a pixel to neighboring pixels. 
As a result, conventional noise filtering methods become less effective in filtering out noise in such a digital Image. 
Furthermore, when the conventional noise-filtering methods are applied directly on the color components prior to CFA 
interpolation, the stencil for the noise filters.must be" widened because of the missing pixels. As a result, extensive 
blurring occurs, especially aroundthe edge-like structures in the image. When color filtering Is applied to an Interpolated 

so image, approximately three times as much processing power Is required to carry out the filtering operation compared 
to performing the filtering before the interpolation. 

[0004] Okisu (U.S. Patent No. 8,091 ,862) discloses a method and device for pixel interpolation in the "up-sampling" 
process to fill In the pixel value of the missing pixels. According to the method of Okisu, when the pixel value of a 
missing pixel in a color component Is calculated for pixel Interpolation, the slopes of pixel values In the same color 
as component around that pixel are used to provide pixel weighting factors in order to Improve the interpolated result. 
While Oktsucan reduce smoothing of edges, its effect is limited because of the missing pixels in the color component 
under pixel Interpolation. 

[0005] Noise filtering a color component using only the pixel values of the same color is known, especially when a 
mean filter or a median filter is used. Figures 1 to 3b are used to illustrate how different noise reduction filters can be 

40 used to filter different color components. Figures 1 to 3b show a section of a color image taken through a Bayer matrix. 
In particular, Figure 2a shows a seotlon of the Bayer matrixed image with relevant pixels used when a green pixel in 
a blue line is filtered. Figure 2b shows a section of the Bayer matrixed Image with relevant pixels when a green pixel 
in a red line is filtered. In both Figures 2a and 2b, the pixel value of the green pixel to be filtered Is denoted by the letter 
O. The pixel values of the surrounding green pixels are denoted by letters A, B, C, D, E, F, G and H, whereas the pixel 

45 values of the surrounding red and blue pixels are denoted by R1-R6, B1-B6. 

[0006] To noise filter the green color component, for example, only the pixel value of green pixels is used. To gen- 
eralize the noise filtering process, It can be said that the green pixel having the pixel value O is filtered by a filter F(), 
and the pixel value of the filtered pixel is denoted by a notation F(0). If the pixels to be used in noise filtering are 
confined to the area as shown in Figures 2a or 2b, then the filter F() Is a combination of pixel values selected only from 

so A, B, C, D, E, F, G and H. The filter F() is not part of the present invention. Thus, F() can be a prior art noise filter or 
any combination of prior art filters, or even any novel filter suitable for noise reduction. 

[0007] Likewise, when a red color component Is subject to noise filtering, only the pixel value of red pixels is used. 
When a blue color component is subject to noise filtering, only the pixel value of the blue pixels is used. Figure 3a 
shows a section of the Bayer matrixed image with relevant pixels when a red pixel is filtered. In Figure 3a, the pixel 

* values of the surrounding red pixels are denoted by letters A, B, C, D, E, F, G and H, whereas the pixel values of the 
surrounding green and blue pixels are denoted by G1-G4 and B1-B4. Figure 3b shows a section of the Bayer matrixed 
image with relevant pixels when a blue pixel Is filtered, In Figure 3b, the pixel values of the surrounding blue pixels are 
denoted by letters A, B, C, D, E, F, G and H, whereas the pixel values of the surrounding green and red pixels are 
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denoted by G1-G4and R1-R4. 

[0008] In priorart, there are two different classes of spatial noise reduction filters used. One is classified as nonlinear 
and the other linear. .A median filter is an example of the non-linear filter, and a mean filter is an example of the linear 
filter. The combination of a non-linear filter and a llnearfilier is another non-linear filter, which can also be used in noise 
filtering. A median filter effectively removes impulsive noise, whereas a mean niter effectively removes Gaussian noise. 
For a human observer, sharp edges are important due lo the properties of the Human Visual System (HVS). Th us, the 
implementation of niters should lake Into account nol only the noise reduction aspect but also the edge preservation 
aspect of the filtering process. Furthermore, low processing power and low memory consumption should also be con- 
sidered and, therefore, thestee of the filter window must be as small as possible. As presented above, combiningthese 
two requirements is a problem in methods of prior art, 

1 ,0 Multistage Median Filter 

[0009] The multistage median filter, as described below, consists of line- and edge-preserving properties, and it still 
effectively reduces noise. The basic components of this multistage median filter consist of four elements: a five-point 
V-median filter, a five-point V^median filter, the original pixel and a three-point median filter of the previous three. 
[0010J Thus, when a pixel having the pixel value O as shown In Figures 2a to 3b, we have: 



?,A,B,C,D) (1.1) 
<"-med - m'edlan5(0, E, F, Q, H) (1 .2) 



(1.3) 



output value = F(O) = medjan3(V-med, "x"-med, original) 



I0O11] This fitter reduces impulse-like noise effectively, and it can be used to attenuate Gaussian noise as well. 
Edges and lines are also effectively preserved. 



[0012] A mean filter usually consists of a structure, where the weighted average of pixels is calculated. Usually, only 
the nearest pixels are used in the fllterwlndow. Thus, the smoothing of edges and lines Is minimal and effective noise 
reduction properties are preserved. Mean filters can be classified into non-directional and directional filters, as de- 
scribed below. 

2.1 Non-dlrectlonal Mean Filter 

[001 3] When green pixels are filtered (Figure 2a or 2b), we can use, for example: 

OUtputG = (4 "O + E + F + G + HyS (2.1.1) 
[0014] When red (or blue) pixels are filtered (Figure 3a or 3b), we can use, for example: 

outputC=(4*0+A + B + C + D)/B (2.1.2) 

[0015] This fitter reduces Gaussian noise effectively and attenuates impulse-like noise. However, being a non-direc- 
tional filter, this mean filter does not effectively preserve llnes and edges. 

2.2 Directional Mean Filter 

[0016] In order to preserve edges and lines more effectively, it is preferred to use directional mean filters as follows: 
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[0017] The green pixels are filtered using, e.g., one of two following filters (Figures 2a and 2b): 



0Utput1G = (2*O + E + H)/4 



(25.1) 



or 



0Utpuf2G = (2*O + F + G)/4. 



(2.2.2) 



[001 8J The red (or blue) pixels are filtered using, for example, one of the two following filters (Figures 3a and 3b): 



[0019] Lowpass spatial filters, similar lo those described hereinabove, and hlghpass spatial filters can be found in 
"Digital Image Processing" by H. c. Gonzalez and R. E. Woods (Addison Wesley Longman, 1993, pp. 1 B9-201). 

Summary of the Invention 

[0020] Based on what has been presented above, it is advantageous and desirable to provide a method and system 
for noise filtering a digital image, wherein the pixel value of a pixel in a color component is adjusted not only based on 
the pixel values of nearby pixels In the same color components, but also based on the pixel values of nearby pixels in 
different: color components. 

[0021 ] The first aspect of the present invention is a method of improving an Image separable into at least a first color 
component and a second color component, wherein the first color component has pixels located at a plurality of first 
pixel locations and a second color component has pixels located at a plurality of second pixel locations. The method 
is characterized by 

noise filtering the first color component using pixel values of the first colorcomponent f or providing a noise-filtered • 
color component having filtered pixel values at the plurality of first pixel locations, and by 

adjusting the filtered pixel value of at. least one first pixel location of the nolse-fittered color component using 
irtfoirnatton indicative of a difference in pixel values of the second color component at the second pixel locations ad- 
jacent to said at least one first pixel location. 

[0022] According to the present invention, the difference In the pixel values is indicative of an edge in the image. 
[0023] According to the present invention, the~ image has a first number of pixels and at least one of the first and 
second color components has at most a second number of pixels smaller than the first number. Preferably, the plurality 
of first pixel locations are different from the plurality of second pixel locations. It is also possible that the first pixel 
locations partially overlap with the second pixel locations. 

[0024] According to the present invention, the image Is separable into three color components of red, green and 
blue. When the first colorcomponent is thegreen component, the second color component Is the red or blue component, 
and when the first color component is the blue or red component, the second colorcomponent is the green component. 
[0025] The second aspect of the present invention is a device for improving an image separable into at least a first 
color component and a second color component, wherein the first colorcomponent has pixels located at a plurality of 
first pixel locations and a second colorcomponent has pixels located at a plurality of second pixel locations. The device 
Is characterized by 

a filtering unit, for noise filtering the first color component using pixel values of the first color component for 
providing a noise-filtered color component having filtered pixel values at the plurality of first pixel locations and for 
providing signals Indicative of the noise-filtered color component, and by 

an adjusting unit, responsive to the signals, for adjusting the filtered pixel value of at least one first pixel location 
of the noise-filtered colorcomponent using information indicative of a difference in pixel values of the second color 
component at the second pixel locations adjacent to said at least one first pixel location. 

[0026] The third aspect of the present invention Is a computer program for improving an image separable into at 
least a first color component and a second color component, wherein the first color component has pixel values located 



output1C = (2*0 + A + B)/4 



(2.2.3) 



0Utput2C = (2 " O + C + D)/4. 



(2.2.4) 
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at a plurality of first pixel locations and the second color component has pixel values located a plurality of second pixel 
locations. The computer program is characterized by 

a first algorithm for noise filtering the first color component using pixel values of the first color component for 
providing a nolse-flltered first color component having filtered pixel values at the plurality of first pixel locations, and by 

a second algorithm, for adjusting the filtered pixel value of ai least one first pixel location of the noise-filtered first 
color component using Information indicative at least of a difference in pixel values of the second color component al 
the second pixel locations adjacent la said at least one first pixel location. 

[0027] The fourth aspect of the present invention is an image processing system having a coior separation device 
for separating an image Into at least a first color component and a second color component for providing a first signal 
indicative of the first color component and a second signal Indicative of the second oblor component, wherein the firat 
coior component has pixels located at a plurality of first pixel locations and the second color component has pixels 
located at a plurality of 6©oond pixel locations. The system is characterized by 

a filtering unit, responsive to the first signal, for noise filtering the first color component.U6lng pixel values of the 
first color component for providing a noise-filtered color component having filtered pixel values at the plurality of first 
pixel locations and for providing a third signal indicative of the noise-filtered color component, by 

an edge detection unit, responsive to the second signal, for providing information indicative at least of a difference 
In pixel, values of the second color component, and by 

an adjustment device,, responsive to the third signal and the information, for adjusting the filtered pixel value of 
at least one first pixel location of the noise-filtered color component based on the difference in pixel values at the 
second pixel locations adjacent to said at least one first pixel location.. • 

The fifth aspect of thep resent Invention Is a method of Improving an image separable into at least aflrat colorcomponent 
and a second color component using a noise filter, wherein the first color component has pixels located at a plurality 
of first pixel locations and a second color component has pixels located at a plurality of second pixel locations, and 
wherein the noise filter comprises a plurality of filter parameters; The method is characterized by 

providing Information indicative of a difference in pixel values of the second color components, and by 
adjusting at least one of the filter parameters in the noise filter for noise filtering the first color component using 
pixel values of the first color component for providing a noise-filtered color component having filtered pixel values at 
the plurality of the first pixel locations, wherein said adjusting is based on the information indicative of the difference 
in pixel values of the second color components at the second pixel locations adjacent to at least one first pixel location 
for providing the filtered pixel value of said at least one first pixel location. 

[0028J The sixth aspect of the present invention is a device for improving an image separable into at least a first 
colorcomponent and a second color component usinga noisetilter, wherein the first colorcomponent has pixels located 
at a plurality of first pixel locations and a second color component has pixels located at a plurality of second pixel 
locations, and wherein the noise filter comprises a plurality of filter parameters for filtering the first color component 
using pixel values of the first colorcomponent for providing a nolse-flltered first coior component having filtered pixel 
values at the plurality of first locations. The device is characterized by a module, capable of modifying the noise filter 
before or during the filtering of the first color component, by adjusting at least one filter parameter using information 
indicative of a difference In pixel values of the second color component at the second pixel locations adjacent to at 
least one first pixel location, for providing the filtered pixel value of said at least one first pixel location. 
[00291 TT»e seventh aspect of the p resent invention is a computer program for improving art Image separable into at 
least a first color component and a second color component using a noise filter, wherein pixels of the first color com- 
ponent are located at a plurality of first pixel locations and pixels of the second color component are located a plurality 
of second pixel locations, and wherein the noise filter comprises aplurality of fitter parameters. The computer program 
Is characterized by 

a first algorithm for noise filtering the first color component using pixel values of the first color component for 
providing a noise-filtered first colorcomponent having filtered pixel values at the plurality of first pixel locations, and by 

a second algorithm for obtaining a difference in pixel values of the second color component at the second pixel 
locations adjacent to at least one first pixel location for adjusting at least one of the filter parameters of the noise filter 
for providing the filtered pixel value of said at least one first pixel location. 

[0030] The eighth aspect of the present invention is an image processing system having a color separation device 
for separating an image into at least a first color component and a second color component for providing a first signal 
indicative of the first colorcomponent and a second signal indicative of the second colorcomponent, wherein the pixels 
of the first color component are located at a plurality of first pixel locations and the pixels of the second color component 
are located at a plurality of second pixel locations. The system is characterized by 

an edge detection unit, responsive to the second signal, for providing information Indicative at least of an edge 
In the image, and by 

a filtering module for noise filtering the first color component for providing a noise-filtered first color component 
having filtered pixel values at the plurality of first pixel locations, the filtering module having a plurality of filter param- 
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eters, at least one of which Isadjusted before or duringthefilteringof the first colorcomponentba6edonthemformation. 
[0031 J According to the present invention, the information Is based on a difference in pixel values of the second color 
component at the pixel locations adjacent to at least one first pixel location and the adjusting is for providing the filtered 
pixel value of the at least one first pixel location. 

[0032] The ninth aspect of the present Invention is a method of improving an image using a noise filter having a 
plurality of filler parameters, wherein the image comprises a sequence of image frames of which at least one frame 
has at least one preceding frame, and each of said at least one frame and said at least one preceding frame is separable 
into at least a first colorcomponent and a second color component, and wherein the first color component has pixels 
located at a plurality of first pixel locations and a second colorcomponent has pixels located at a plurality of second 
pixel locations. The method is characterized by 

providing Information indicative of a difference in pixel values of the second color components of said at least 
one preceding frame, and by 

adjusting at least one of tho filter parameters in the noise ftlterlor noise filtering the nret color component of said 
at least one frame using pixel values of the first color component for providing a noise-filtered color component having 
filtered pixel values at the plurality of the first pixel locations, wherein saldadjusttng is based on the information indicative 
of the difference in pixel values of said at least one preceding frame at the second pixel locations adjacent to at least 
one first pixel location for providing the filtered pixel value of said at least one first pixel location. 
[0033] The tenth aspect of the present Invention is a device for improving an image using a noise filter, wherein the 
image comprises a sequence of image frames of which at least orie frame has at least one preceding frame, and each 
of said at least one frame and said at least one preceding frame Is separable Into at least a first color component and 
a second color component, and the first color component has pixels located at a plurality of first pixel locations and a 
second colorcomponent has pixels located at a.plurallty of second pixel locations, and wherein the noiBe filter comprises 
a plurality of fiker parameters for filtering the first color component of said at least one frame using pixel values of the 
first color component for providing a noise-filtered color component having filtered pixel values at the plurality of first 
pixel locations. The device is characterized by 

a module, capable of modifying the noise filter before or during the filtering ol the first color component of said 
at least one frame, by adjusting at least one of the filter parameters in the noise filter using information indicative of a 
difference in pixel values of the second color component of said at least one preceding frame at the second pixel 
locations adjacent to at least-one first pixel location, for providing a modified noise filter for filtering the first color 
component of said at least one frame for providing the filtered pixel value of said at least one first pixel location. 
[0034] The present invention will become apparent upon reading the description taken In conjunction with Figures 
1 to 9b. 

Brief Description of the Drawings 



Figure 1 is a diagrammatic representation showing a section of a color image obtained through a Bayer matrix. 
Figure 2a is a diagrammatic representation showing a section of the color Image when a gre B n pixel in a blue line 
is filtered. 

Figure 2b Is a diagrammatic representation showing a section of the color Image' when a green pixel In a red line 
is filtered. 

Figure 3a is a diagrammatic representation showing a Section of the color image when a red pixel Is filtered. 
Figure 3b is a diagrammatic representation showing a section of the color Image when a blue pixel Is filtered. 
Figure 4 is a diagrammatic representation showing a macroplxel. 

Figure 5 is a block diagram showing a system for noise filtering a color image, according to the present invention. 
Figure 6 is a flow chart showing a method for noise filtering a color Image, according to the present invention. 
Figure 7 is a block diagram showing a system for noise filtering a color Image, according to another embodiment 
of the present invention. 

Figure 8 is a flow chart showing a method for noise filtering a color image, according to another embodiment of 
the present invention. 

Figure 9a is a matrix representing a noise filter with a plurality of filter parameters. 

Figure 9b is a matrix representing a noise filter wherein one of the filter parameters is adjusted based on edge 
information. 

Best Mode to Carry Out the Invention 



[0036] There are two embodiments of the present invention, depending on the characteristics of the Bayer matrixed 
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image to be filtered. A Bayer matrixed image can be considered as an image consisting of a plurality of macropixels 
containing four original pixels, as shown in Fig. 4. When there is no significant difference between picture elements 
G 1 and G2, in terms of the elements themselves, then a first embodiment is preferred. Otherwise, a second embodiment 
is preferred. Itshould be noted that the difference between 61 and G2 at a location can arise from a number of factors. 
For example, It can be the result of the Bayer color filter, the sensitivity and response of the pixels, the amplification 
or the pixel response and so forth, although the color of the object at that location is substantially, the same. However, 
the difference resulted from these factors is usually quite insignificant. 

A. First Preferred Embodiment 

, [0037] Trio image is processed using a multistage median filter, as described In subsection (1 .0). The algorithm for 
the first ombodimont involves two threshold values, Thr1 and Thr2. Thr1 and Thr2 are noise-dependent values. It is 
preferred that Thr1=8 and Thr2-1 2, but they can be set equal to different values. The values of the thresholds can 
even be different in different parts of the image. It is assumed here that Urn pixel values can be expressed with 8 bits, 
but the bit depth of the pixel values can be any other value, as well, if the bit depth is different from 8 bits, the threshold 
values can be changed accordingly. 

A.1 Noise Filtering of Green Color Component 

[0038] When the green pixel is filtered (Figure 2a or 2b), the algorithm Is as follows: 



absolute value of (Rl - R2) is greater than Thr 1 
or . 

absolute value of (Bl - B2) is greater than Thrl 

then 

the output value is clipped to values between (Original - Thr2) and 
(Original + Thr2). 

Otherwise 

the output value is not clipped. 

where 

Original - O 

output value = median3("+"-med, "x"-med, Original) (see Subsection 1.0) 



K.2. Noise filtering of Red or Blue Color Component 

[0039] When a red (or blue) pixel is filtered (Figure 3a or 3b). the algorithm Is as follows: 
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If 

absolute value of (Gl .- G4) is greater than Thrl 

or 

absolute value of (G2 - G3) is greater than Thrl 

then 

the output value is clipped to values between (Original - Thr2) and 
(Original + Thr2). 
Otherwise 

the output value is not clipped. 

where 

Original » O 

output value = mediao3("+"-med, "x"-raed, Original) (see Subsection 1.0) 
B. Second Preferred Embodiment ... 

[0040]. The second preferred embodiment can be descrtoed as having two steps. The first step involves the calcu- 
lation of a temporary output value of the fitter to be used, and the second step involves the clipping of the temporary 
output value. 

B.1 Noise Filtering of Green Color Component 

[0041 ] When a green color component Is filtered, It is preferred that the threshold values are Thrl = 6, thr2 = 4 and 
thr3 - 28, but they can be set equal to different values. The values of the thresholds can even be different in different ' 
parts of the image. It is assumed here that the pixel values can be expressed with 8 bits, but the bit depth of the pixel 
values can be any alrmr value. If the bit depth is different from 8 bits, the threshold values can be changed accordingly. 
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B,1 .1 Calculation of temporary output value of filter 
{0042] 

If 

abs((E + H>(F + G)) is smaller than or equal to Thrl 

then 

TEMPG ^outputG (see Subsection 2.1) 
otherwise 
if 

SUM(E-H)<SUM(F-G), 

then 

TEMPG -outputlG 
otherwise 

TEMPG - ou1put2G 



SUM(E - H) = abs(2 * O - (E + H)) + 2 * abstE - H) + abe((B1 + B3) - (82 + BS))/2 + 

90 

abs((R1 + R3) - (R2 + RB))y2. 

SUM(F-H) = abs<2* O - (F + G)) + 2* abs(F - S) + abs((B1 + B5) ■ (B2 + B4))/2 + 
abs{(R1 + R5) - (R2 + R4»/2. 
[0043J Above, the notation abs() indicates an absolute value. 
*> B.1 .2 Clipping of the temporary output value of filter 
[0044] 



abs(Original - TEMPG) is larger than thr2 

then 

If 
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MADDG > Thr3 

then 

LIMIT = Thr2 
otherwise 

LIMIT = Thr3 - MADDG +Thr2 
Output value = TEMPG clipped to a value between (Original - LIMIT) and 
(Original + LIMIT) 
otherwise 

Output value = TEMPG (not clipped). 

where 

MADDG =MAX(abs(R1 - R2), abs(B1 - B2), abs(E - H), abs(F - G),abs(A 
-By2,abs(C-D)/2). 

10045] Above, the notation abs() Indicates an absolute value and MAX(x,y,z) Indicates the largest of x, y and z. 
B.2 Noise Filtering of Red and Blue Color Components 

[0046] When a red or blue color component Is Altered, It Is preferred that Thrl is 4, thr2 is 3 and thr3 is 29. • 

B.2.1 Calculation of temporary output value of filter 

[00471 

If 

abs((A + B)-(C + D)) is smaller than or equal to Thrl 

then 

TEMPC = outputC (see Subsection 2.1) 
otherwise 
if 

SUM(A - B)<SUM(C - D), 

then 

TEMPC = outputlC 

otherwise 

TEMPC = output2C 
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Where: 

SUM(A - B) = abs(2 * O - (A + B)) + 2 * abs(A - B) + abs(G2 - G3) 

SUM(C - D) = abs(2* 0-{C + D)) + 2* abs(C - D) + abs(Q1 - G4) 
[0048] Above, the notation absQ indicates an absolute value. 
B.2.2 Clipping of the temporary output value of filter 
[0049] 

If 

abs(Original - TEMPC) is larger than Thr2 

then 

If 

MADDC > Thr3 

then 

LIMIT = Thr2 
otherwise 

LIMIT ~ Thr3 - MADDC + Thr2 
Output value - TEMPG clipped to a value between (Original - LIMIT) and 
(Original + LIMIT) 
otherwise 

Output value = TEMPG (not clipped). 

Wherein 

If directional filter is used in A-B direction, then 

MADDC * MAX(abs(61 - 64), abs(G2 - G3), abs(B1 - B4), abs(B2 - B3), abs(A - B)/2), for the red 
color component, and MADDC = MAX(abs(G1 - G4), abs(G2 ■ G3), abs(R1 - R4), abs(R2 - R4), abs(A - B)/2), forthe 
blue color component; 
and 

If directional filter is used in C-D direction, then 

MADDC = MAX(abs(G1 - G4). abs(G2 - G3), abs(B1 - B4), abs(B2 - B3), abs(C - D)/2), forthe red 
color component, and MADDC = MAX(abs(G1 - G4), abs(G2 - G3), abs{R1 - R4), abs(R2 - R3), abs(C - D)/2), for the 
blue color component; 
and 

If non-directional filter Is used 

MADDC =MAX(abs(GI - G4), abs(G2-G3), abs(B1 - B4), abs(B2 - B3), abs(A-B)/2, abs(C-D)/2), for 
the red color component, and MADDC = MAX(abs(G1 - G4), abs(G2-G3), abs(R1 - R4), abs(R2 - R3), abs(A-B)/2, abs 
(C-D)/2), for the blue color component. 

[0050] Above, the notation abs() indicates an absolute value, and MAX(x,y,z) indicates the largest of x, y and z. 
[0051 ] It should be noted that the values of Thr1 , Thr2 and Thr3 are exemplary values for carrying out the preferred 
embodiments as described in Section A and Section B. These, values can be different. 

[0052] To facilitate the improvement of a raw Bayer matrlxed Image, It Is preferred that an Image processing system, 
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as shown in Figure S, is used. The image processing system 5 comprises a sub-system 30 and an edge detection unit 
40. The sub-system 30 comprises a noise littering unit 32 and an adjustment unit 34. As shown In Figure 5, the raw 
Bayer matrixed mage 110, which Is provided by an imaging device 10, is separated by a color separation unit 20 into 
three color components represented by signals 1 21 , 122 and 1 23. For example, the signal 121 is indicative of the green 
color component. As the signals 121 , 1 22, 123 are conveyed to the system 5, they are processed by the noise filtering 
unit 32 to become noise-mtered color components, represented by signals 131, 132, 133. At the same time, the edge 
detection unit 40 extracts from the signals 121, 122 and 123 information indicative of edges or lines In the image 110. 
The extracted information is represented by signals 141,142, 143. For example, if the green colorcomponent Is filtered 
by the noise filtering unit 32, responsive to the signal 1 21 , then the Information 142 or143 will be used by the adjustment 
unit 34 to adjust the pixel values Ifi the noise-filtered color component represented by the signal 131 . After all three 
noise-filtered color components are adjusted by the adjustment unit 34, they are provided, as signals 151, 152 and 
153 to a CFA unit for CFA interpolation. 

[0053] It should be noted that all the components 10, 20, 32, 34, 40 and 50 can be part of an imaging system, but 
they can be disposed in two or more physically unconnected systems. For example, the image 110 from the imaging 
device 10 can be conveyed to the color separation unit 20 via a wired connection or In a wireless manner. Likewise, 
the color components from the color separation unit 20 can be conveyed to the image processing system 5 through 
the Internet, via a recording medium or using infrared signals. Furthermore, the system 5 can be implemented as a 
single device having a single computer program to carry out the tasks of noise tittering, edge detection and pixel value 
adjustment. 

[0054] The method of improving an image, according to the present invention, can be summarized In a flowchart, 
as shown in Figure 6. As shown in Figure 6, the method 500, starts at step 510 where the signals 121, 122 and 123 
indicative of the three color components are received by the image processing system 5. Responsive to the received 
signals, the noise filtering unit 32 filters the color components at step 512 for providing the noise-filtered color compo- 
nents and the edge detection unit 40 detects edges in the color components at step 514. If edges are found, as deter- 
mined at step 516, then the pixel values of the pixels in one color component located near the edges in the other color 
components are adjusted by the adjustment unit 34 at step 518. At step 520, the color components, as processed by 
the noise filtering unit 32 and adjusted by the adjustment unit 34 based Dn edge information provided by the edge 
detection unit 40, are conveyed, directly or indirectly, to a CFA unit for CFA Interpolation. 
[0055] Figure 7 shows an Image processing system 5', according to another embodiment of the present invention. 
As shown, the image processing system 5' comprises a subsystem 30' and an edge detection unit 40. The subsystem 
30* comprises a noise filtering unit 32' that uses a noise Biter (see Figure 9a) having a plurality of filter parameters and 
a parameter adjustment module 36. As th e edge detection unit 40 extracts from the signals 1 21 , 1 22 and 1 23 information 
indicative of edges or lines In the Image 110, the edge detection unit 40 provides the extracted Information 141, 142, 
143 to the subsystem 30". Based on the extracted Information 141, 142, 143, the parameter adjustment module 36 
adjusts the filter parameters of the noise Alter in the noise filtering unit 32'. For example, one of the filler parameter 
is adjusted by using the Information from the adjustment module 36. The modified noise filter Is shown In Figure 9b. 
Alternatively, If the noise filter is a median filter, the properties of the median filter (like the pixels whose values are 
used for filtering) can be altered by using the information from the adjustment module 36. 
[0056] The method 600 of Improving an image, according to the other embodiment, Is shown in Figure B. As shown, 
the method 600 starts at steps 610 and 620, wherein signals indicative of the three color components are received in 
the Image processing system 5' and a noise filtering unit 32' based on a noise filter (Figure 9a, for example) having a 
plurality of f I Iter parameters is used for filtering the three color components. The edge detection unit 40 Is used to detect 
edges In the color components at step 630. If edges are found, as determined at step 640, then one or more of the 
filter parameters is adjusted at step 650 to provide a modified noise filter (Figure 9b, for example). The modified noise 
filter is used to filter the color components at step 660. Otherwise, the original filter (Figure 9a) is used at step 660. 
[0057] In connection with Figures 7 and 8, it should be understood that the filter parameters of the unit 32' can be 
indicative of the filter parameters to be used, such as presented In Figures 9a and 9b, or they can be indicative of the 
pixels whose values are to be used in filtering. They can even be indicative of the filter orf ilter type to be used In filtering. 
[OOSB] Alternatively, the edge Information provided by the edge detection unit 40 is used to adjust the pixel values 
of the pixels near the locations of the edges, and then an unadjusted noise filter is used to filter all pixels. In that way, 
the pixels are pre-filtered using the edge information in order to provide pixel value compensation In foreseeable noise 
filtering. 

[0059] The method and system for improving an image, according to the present invention, have been described In 
the context of a Bayer matrixed Image, wherein the pixel locations of one color component and the pixel locations of 
other colorcomponents do not overlap. In otherwords, a pixel in the green colorcomponent Is located In a pixel location 
different from the pixel location occupied by any pixel in the red or blue colorcomponent. However, the same method 
and system can be used to process color components that have overlapping pixel locations so long as the adjustment 
of the noise filtered pixel values In one color component Is based on the edge information extracted from one or two 
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other color components. Moreover, the algorithm for noise filtering of color components are lowpass spatial filters, 
which can be based on linear spatial Alters, non-linear filters or the combinations thereof. The algorithm for edge 
detection can be based on direction or non-direction spatial filters. 

[0O6OJ The invention is based on the observation that in many images the locations of edges across different color 
components are highly correlated, i.e., the edges are present in all color components, whereas the noise is not corre- 
lated across the color components, especially In systems where the pixels of different color components are not co- 
localed. 

[0061] The method, according to the present invention, can be applied to 3-tSmenslonal images , as well . For example, 
in medical imaging there might arise a situation where one image component Is present at certain locations and another 
at different locations, i.e., they are different modalities of the image. Then, edge Information (whether 2-D or 3-D) is 
used to adjust the noise Altering (2-D or 3-D). Other examples may be radar Images and images taken from material 
Structure, etc. The method can also be applied to video Images so that the edges are detected from one image (pref- 
erably the l-frame) and the noise filtering (if carried out) Is controlled by this information in multiple hiages. The noise 
filtering process may also comprise averaging over a few images, If there is a static area In the image. Of course, noise 
filtering cannot be controlled by an edge from a different frame In a moving part of the image being filtered. 
[0062] Furthermore, the present invention Is applicable to an imaging system using different wavelengths to acquire 
an image. These wavelengths can be in the radio frequency range, in the infrared or ultraviolet spectral regions. 
P0631 Thus, although the invention has been described with respect to a preferred embodiment thereof, it will be 
understood by those skilled in the art that the foregoing and various other changes, omissions and deviations in the 
form and detail thereof may be made without departing from the scopB of this Invention. 



Claims 

1. A method of Improving an image separable into at least a first color component and a second color component, 
the first color component having pixels located at a plurality of first pixel locations and a second color component 
having pixels located at a plurality of second pixel locations, said method characterized by 

noise filtering the first color component using pixel values of the first color component for providing a nolse- 
f lltered color component having filtered pixel values at the plurality of first pixel locations, and by 

adjusting the filtered pixel value of at least one first pixel location of the noise-filtered color component using 
information indicative of a difference in pixel values of the second color component at the second pixel locations 
adjacent to said at least one first pixel location . ' 

2. The method of claim' 1 , characterized In that the difference in the pixel vakies.is Indicative of an edge in the Image. 

3. The method of daim' 1 , characterized I n that the image has a first number of pixels and at least one of the first 
and second color components has' at most a second number or pixels smaller than the first number. 

4. The method of claim 1 , characterized In that the plurality of first pixel locations are different from the plurality of 
second pixel locations. 

5. The method of claim 1 , characterized in that the Image is separable Into three color components in red, green 
and blue, and that the first coior component is the red component and the second color component is the green 
component. 

6. The method of claim 1 , characterized In that the Image is separable into three color components in red, green 
and blue, and that the first color component is the blue component and the second color component is the green 
component. 

7. The method of claim 1 , characterized In that the image Is separable into three color components in red, green 
and blue, and that the first color component is the green component and the second color component Is the red 
component. 

8. The method of claim 1 , characterized in that the image is separable into three color components in red, green 
and blue, and that the first color component is the green component and the second color component is the blue 
component. 

9. The method of claim 1 , characterized in that the image is a color image acquired from a sensor array having a 



13 



P 1315M7A2_L> 



EP1 315 367 A2 



Bayer matrix disposed thereon for color f (tiering. 

10. A device for Improving an image separable into at least a first color component and a second color component, 
the first color component having pixels located at a plurality of first pixel locations and the second color component 
having pixels located at a plurality of second pixel locations, said device characterized by 

means, for noise tillering the first color component using pixel values of the first coloroomponentforprovidlng 
a noise-filtered color component having filtered pixel values at the plurality of first pixel locations and for providing 
signals indicative of the noise-filtered color component, and by 

■ means, responsive to the signals, for adjusting the filtered pixel value of at least one first pixel location of 
the noise-f lltered color component using information Indicative at least of a difference in pixel values of the second 
color component at the second pixel locations adjacent to said at least one first pixel location. 

1 1 . The device of claim 1 0, characterized In that the difference is indicative of an edge in the Image. 

12. The device of claim 10, characterized in that the image has a first number of pixels and at least one of the first 
and second color components has at most a second number of pixels smaller than the first number. 

13. The device of claim 10, characterized in that the plurality of first pixel locations are different from the plurality of 
second pixel locations. 

14. The device of claim 10, characterized in that tfie Image is separable Into three color components of red, green 
and blue. 

15. A computer program for improving an Image separable Into at least a first color component and a second color 
component, the first color component having pixel values located at a plurality of first pixel locations and the second 
color component having pixel values located at a plurality of second pixel locations, said computer program char- 
acterized by 

a first algorithm for noise filtering the first colorcomponent using pixel values of the first color component for 
providing a noise-filtered first color component having filtered pixel values at the plurality of first pixel locations, 
and by 

a second algorithm, for adjusting the filtered pixel value of at least one first pixel location of the noise-filtered 
first color component using information Indicative at least of a difference In pixel values of the second color com- 
ponent at the second pixel locations adjacent to said at least one first pixel location. 

16. The computer program of claim 15, characterized in thatthe difference in pixel values is Indicative of an edge In 
the image. 

17. The computer program of claim 1 5, characterized In that the plurality of first pixel locations are different from the 
plurality of second pixel locations. 

18. The computer program of claim 15, characterized in thatthe image is separable into three color components of 
red, green and blue.' 

19. An image processing system having a color separation module for separating an image Into at least a first color 
component and a second color component for providing a first signal indicative of the first color component and a 
second signal indicative of the second color component, wherein the first colorcomponent has pixels located at 
a plurality of first pixel locations and the second color component has pixels located at a plurality of second pixel 
locations, said system characterized by 

a filtering unit, responsive to the first signal, for noise filtering the first color component using pixel values of 
the first color component for providing a noise-filtered color component having filtered pixel values at the plurality 
of first pixel locations and for providing a third signal indicative of the noise-filtered color component, by 

an edge detection unit, responsive to the second signal, for providing Information indicative at least of a 
difference In pixel values of the second color component, and by 

an adjustment device, responsive to the third signal and the information, for adjusting the filtered pixel value 
of at least one first pixel location of the noise-filtered color component based on the difference in pixel values of 
the second color component at the second pixel locations adjacent to said at least one first pixel location. 



20: The system of claim 1 9. characterized by the difference in pixel values Is Indicative of an edge in the image-. 
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21. A method of Improving an image separable into at least a first color component and a second color component 
using a noise filter, wherein the first uuloi component has pixels located at 'a plurality of first pixel locations and a 
second color component has pixels located at a plurality of second pixel locations, and wherein the noise filter 
comprises a plurality of lilter parameters, said method characterized by 

providing Information Indicative of a difference in pixel values of the second color components, and by 
adjusting at least one of the filter parameters in the noise filter for noise filtering the first color component 
using pixel values of the first color component for providing a noise-filtered color component having filtered pixel 
values at the plurality of the first pixel locations, wherein said adjusting is based on the Information indicative of 
the difference In pixel values of the second color components at the second pixel locations adjacent to at least 
one first pixel location for providing the filtered pixel value of said at least one first pixel location. 

22. The method of claim 21 , characterized in thattho difference In pixel values is Indicative of an edge In the image. 

23. A device for improving an image separable Into at least a first color component and a second color component 
using a noise filter, wherein the first color component has pixels located at a plurality of first pixel locations and a 
second color component has pixels located at a plurality of second pixel locations, and wherein the noise filter 
comprises a plurality of filter parameters for filtering the first color component using pixel values of the first color 
component for providing a noise-filtered first color component having filtered pixel values at the plurality of first 
pixel locations, said device characterized by 

a module, capable of modifying the noise filter before or during the filtering of the first color component, by 
adjusting at least one filter parameter using information Indicative of a difference in pixel values of the second color 
component at the second pixel locations adjacent to at least one first pixel location, for providing the filtered pixel 
value of said at least one first pixel location. 

24. The device of claim 23, characterized In that the difference in pixel values is indicative of an edge in the image. 

25. A computer program for Improving an image separable into at teast a first color component and a second color 
component using a noise filter, wherein pixels of the first color component are located at a plurality of first pixel 
locations and pixels of the second color component are located at a plurality of second pixel locations, and wherein 
the noise filter comprises a plurality of filter parameters, said computer program characterized by 

a first algorithm for noise filtering the first color component using prxel values of the first color component for 
providing a noise-filtered first color component having filtered pixel values at the plurality of first pixel locations, 
and by 

a second algorithm for obtaining a difference In pixei values Df the second color component at the second 
pixel locations adjacent to at least one first pixel location for adjusting at least one of the filter parameters of the 
noise filter for providing the filtered pixel value of said at least one first pixel location. 

26. An image processing system having a color separation module for separating an Image Into at least a first color 
component and a second color component for providing a first signal Indicative of the first color component and a 
second signal indicative of the second color component, wherein the pixels of the first color component are located 
at a plurality of first pixel locations and the pixels of the second color component are located at a plurality of second 
pixel locations, said system characterized by 

an edge detection unit, responsive to the second signal, for providing Information Indicative at least of an 
edge in the image, and by 

afiltering modulefornolsefilteringthefirsteolorcomponentforprovidinga noise-filtered first color component 
having filtered pixel values at the plurality of first pixel locations, the filtering module having a plurality of filter 
parameters, at least one of which is adjusted before or during the filtering of the first color component based on 
the information. 

27. The system of claim 28, characterized In that the Information is based on a difference in pixel values of the second 
color component at the second pixel locations adjacent to at least one first pixel location and said adjusting is for 
providing the filtered pixel value of said at least one first pixel location. 

28. A method of Improving an image using a noise filter having a plurality of filter parameters, wherein the Image 
comprises a sequence of image frames of which at least one frame has at least one preceding frame, and each 
of said at least one frame and said at least one preceding frame is separable Into at least a first color component 
and a second color component, and wherein the first color component has pixels located at a plurality of first pixel 
locations and a second color component has pixels located at a plurality of second pixel locations, said method 
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characterized by ■ 

providing informatfeh Indicative of a difference in pixel values of the second color components of said at least 
one preceding frame, and by 

adjusting at least one of the filter parameters in the noise filter for noise filtering the first color component of 
said at least one frame using pixel values of the first colo r component for providing a nolse-f lltered color component 
having filtered pixel values at the plurality of the first pixel locations, wherein said adjusting Is based on the infor- 
mation indicative of the difference in pixel values of said at least one preceding frame at the second pixel locations 
adjacent to at least one first pixel location for providing the filtered pixel value of said at least one first pixel location. 

29. The method of claim 28, characterized In that the difference in pixel values is indicative of an edge in the image. 

30. The method of claim 28, characterized in that the image is a video Image. 

31. A device for improving an Image using a noise filter, wherein the image comprises a sequence of image frames 
of which at least one frame has at least one preceding frame, and each of said at least one frame and said at least 
ona preceding frame Is separable Into at least a first color component and a second color component, and the first 
color component has pixels located at a plurality of first pixel locations and a second color component has pixels 
locatedat a plurality of second pixel locations, and wherein the noise filter comprises a plurality of filter parameters 
for filtering the first color component of said at least one frame using pixel values of the first color component for 
providing a noise-filtered color component having filtered pixel values at the plurality of first pixel locations, said 
device characterized by 

a module, capable of modifying the noise filter before or during the filtering of the first color component of 
said at least one frame, by adjusting at leastone of the filter parameters In the noise fllterusinginformation indicative 
of a difference In pixel values of the second color component of said at least one preceding frame at the second 
pixel locations adjacent to at least one first pixel location, for providing a modified noise filter for filtering the first 
colorcomponent of said at leastoneframe for provldlngtheflitered pixel valueof said atleastonefirstpixel location. 
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